Delegate conditional-decode 412 decision to prefix-based-pd-decider by dmitripikus · Pull Request #1684 · llm-d/llm-d-router

dmitripikus · 2026-06-18T11:29:39Z

What type of PR is this?

/kind feature

What this PR does / why we need it:
Move the RFC 7240 Prefer: if-available 412 gate out of the director and into
the prefix-based-pd-decider plugin. One nonCachedTokens threshold now drives
both decisions:
disaggregated P/D and conditional-decode rejection.

The director had simplistic and non-configurable implementation of decode condition.

Behavior change: the 412 gate now rejects when the non-cached suffix is at or above nonCachedTokens (rather than rejecting only on a complete cache miss), and nonCachedTokens: 0 disables the gate entirely. Deployments wanting the 412 gate must include - type: prefix-based-pd-decider with a non-zero nonCachedTokens in their EPP config; deployments using the canonical pd-epp-config.yaml already declare the plugin and pick up the gate automatically. See docs/disaggregation.md for details.

Which issue(s) this PR fixes:

1686

Changes

New requestcontrol.ConditionalDecodeDecider interface; auto-registered
through Config.AddPlugins.
*PrefixBasedPDDecider implements it via ShouldRejectConditionalDecode,
sharing computeNonCachedTokens with the existing disaggregate path
(different fail-mode semantics: 412 fail-closed, disaggregation fail-open).
Director consults the configured decider

Release note

    plugins:
     ...
     - type: prefix-based-pd-decider
      parameters:
        nonCachedTokens: <enter tokens number here>
    ...
    
 should be added to decoder configuration map to support conditional decode

Signed-off-by: Dmitri Pikus <DPIKUS@il.ibm.com>

roytman · 2026-06-18T12:37:24Z

Please open an issue and add it to the PR,

Behavior change in the 412 gate needs to be called out explicitly. This is the substantive change, and it is easy to miss:

Old gate (primaryEndpointHasCachedPrefix): forwarded if the endpoint had any matching prefix block (MatchBlocks() > 0); rejected only on a complete cache miss.
New gate: rejects when the non-cached suffix is >= nonCachedTokens, and when nonCachedTokens == 0 the gate is disabled entirely (always forwards).

We need to update the README file about the conditional decode behaviour.

…f/else" Signed-off-by: Dmitri Pikus <DPIKUS@il.ibm.com>

Signed-off-by: Dmitri Pikus <DPIKUS@il.ibm.com>

roytman · 2026-06-18T12:08:55Z

-// a real cache miss.
-func primaryEndpointHasCachedPrefix(logger logr.Logger, result *fwksched.SchedulingResult) bool {
+// primaryDecodeEndpoint returns the first endpoint chosen by the primary
+// profile, or nil when the result is empty or malformed. False-return reasons


I think the comment is stale.
The function no longer inspects the cache, it only returns the endpoint. So, the "cache miss" framing is now inaccurate.

Fixed, Thank you!

roytman · 2026-06-18T12:12:10Z

-				Code: errcommon.PreconditionFailed,
-				Msg:  "no decode worker has the requested KV cache",
+		decider := d.requestControlPlugins.ConditionalDecodeDecider()
+		switch {


if decider == nil { logger.V(logutil.DEBUG).Info("conditional-decode: no decider configured, forwarding") } else { endpoint := primaryDecodeEndpoint(logger, result) if endpoint == nil || decider.ShouldRejectConditionalDecode(ctx, reqCtx.SchedulingRequest, endpoint) { ... } ... }

I've resolved this when taking care of lint problem.
Thanks!

…hange Signed-off-by: Dmitri Pikus <DPIKUS@il.ibm.com>

dmitripikus · 2026-06-18T15:33:20Z

@roytman , addressing your fist set of comments:

issue is created and associated with PR
behavior change is clarified in "README.md" you mentioned, and also in "disaggregation.md" and in PR description.
Thanks!

vMaroon · 2026-06-18T15:33:43Z

+- Plugin not declared in the config → gate disabled (the EPP forwards conditional-decode requests unconditionally).
+- Prefix-cache state unreadable on the chosen endpoint (e.g. no approximate-prefix producer wired up) → fail closed (412).
+
+This behavior differs from earlier releases, where the gate was a hard-coded check inside the director that forwarded on any cache hit and rejected only on a complete cache miss. The threshold is now in tokens (not blocks), is operator-configurable, and is opt-in via the plugin.


We should avoid temporal language as instructed in AGENTS.md.

Would be interesting to understand if your agent "willingly" disobeyed that instruction or it just lost it in context.

@vMaroon , this is an interesting point.
In this case the agent didn't work according to the instruction.
The agent states that "it didn't deliberately set the rule aside, but it also didn't consciously check it against the wording I was producing..."
We probably need to explicitly re-check/re-run against AGENTS.md

In any case, I re-wrote the comment not to contain temporal language

vMaroon · 2026-06-18T15:34:53Z


+**Conditional-decode 412 gate**
+
+When this plugin is configured, it also drives the EPP's RFC 7240 `Prefer: if-available` (conditional-decode) gate. The coordinator uses that header to mark a speculative early-decode attempt: forward to a decode worker only if its KV cache already covers the prompt, otherwise return HTTP 412 Precondition Failed so the coordinator restarts the pipeline at encode/prefill/decode.


Wonder if this gate can be described independently, with coordinator use as a note.

I re-wrote the comments to describe the gate independently and pull the coordinator behavior into a note. Thank you!

vMaroon · 2026-06-18T15:35:45Z

+// given the chosen decode endpoint, should the request be rejected with HTTP
+// 412 (because the local KV cache does not cover enough of the prompt) or
+// forwarded? At most one such plugin is consulted per director.
+type ConditionalDecodeDecider interface {


Description should be self-contained, and descriptive of what the module does.

I re-phrased the description.

Thank you!

Signed-off-by: Dmitri Pikus <DPIKUS@il.ibm.com>

github-actions · 2026-07-10T01:41:17Z

This PR is marked as stale after 21d of inactivity. After an additional 14d of inactivity (7d to become rotten, then 7d more), it will be closed. To prevent this PR from being closed, add a comment or remove the lifecycle/stale label.

Delegate conditional-decode 412 decision to prefix-based-pd-decider

dbfce5b

Signed-off-by: Dmitri Pikus <DPIKUS@il.ibm.com>

dmitripikus requested a review from a team as a code owner June 18, 2026 11:29

dmitripikus requested review from ahg-g and liu-cong June 18, 2026 11:29

github-actions Bot added kind/feature Categorizes issue or PR as related to a new feature. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 18, 2026

github-actions Bot added kind/feature Categorizes issue or PR as related to a new feature. and removed kind/feature Categorizes issue or PR as related to a new feature. labels Jun 18, 2026

git commit -sa -m "fix(epp): flatten conditional-decode switch into i…

7bd1128

…f/else" Signed-off-by: Dmitri Pikus <DPIKUS@il.ibm.com>

github-actions Bot added kind/feature Categorizes issue or PR as related to a new feature. and removed kind/feature Categorizes issue or PR as related to a new feature. labels Jun 18, 2026

docs: document conditional-decode 412 gate behavior change

541981e

Signed-off-by: Dmitri Pikus <DPIKUS@il.ibm.com>

roytman reviewed Jun 18, 2026

View reviewed changes

docs: more clarification about conditional-decode 412 gate behavior c…

cd229c3

…hange Signed-off-by: Dmitri Pikus <DPIKUS@il.ibm.com>

vMaroon reviewed Jun 18, 2026

View reviewed changes

dmitripikus added 4 commits June 18, 2026 18:41

git commit -s -m "docs(epp): fix stale comment on primaryDecodeEndpoint"

4cf1121

Signed-off-by: Dmitri Pikus <DPIKUS@il.ibm.com>

Comment is changed not to contain temporal language

765591d

Signed-off-by: Dmitri Pikus <DPIKUS@il.ibm.com>

docs: describe conditional-decode gate independently of coordinator

8f5e2e6

Signed-off-by: Dmitri Pikus <DPIKUS@il.ibm.com>

docs(epp): make ConditionalDecodeDecider doc self-contained

4035835

Signed-off-by: Dmitri Pikus <DPIKUS@il.ibm.com>

github-actions Bot added the lifecycle/stale label Jul 10, 2026


		Conditional-decode 412 gate

		When this plugin is configured, it also drives the EPP's RFC 7240 `Prefer: if-available` (conditional-decode) gate. The coordinator uses that header to mark a speculative early-decode attempt: forward to a decode worker only if its KV cache already covers the prompt, otherwise return HTTP 412 Precondition Failed so the coordinator restarts the pipeline at encode/prefill/decode.

Uh oh!

Conversation

dmitripikus commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

roytman commented Jun 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmitripikus commented Jun 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dmitripikus commented Jun 18, 2026 •

edited

Loading